feat(ci,docs): build and publish a combined HTML and spec doc artifact#2638
feat(ci,docs): build and publish a combined HTML and spec doc artifact#2638danceratopz merged 27 commits intoforks/amsterdamfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #2638 +/- ##
================================================
Coverage 88.17% 88.17%
================================================
Files 577 577
Lines 35659 35659
Branches 3490 3490
================================================
Hits 31442 31442
Misses 3654 3654
Partials 563 563
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
6485dc5 to
4dcd1d9
Compare
|
Successful artifact publish here: https://github.com/ethereum/execution-specs/actions/runs/24453695866 |
Split HTML docs (`mkdocs`) and spec docs (`docc`) into parallel CI jobs that produce a combined artifact for `steel-website` deployment. - Add `docs-build.yaml` with five jobs: `check-should-publish`, `html-docs`, `spec-docs`, `combine`, and `trigger-aggregator`. - Add `.github/configs/docs-branches.yaml` branch allowlist. - Slim `test-docs.yaml` to changelog and markdown lint only (renamed to "Static Doc Checks"); the `mkdocs` build job moves to the new workflow. - Update `mkdocs.yml` `site_url` to read from `SITE_URL` env var with fallback to `steel.ethereum.foundation/docs/`. - Update `DocsConfig` URLs to `steel.ethereum.foundation/docs`.
Add `experiments/**` branch trigger and `experiments/publish-docs` to the allowlist so the full pipeline (including `combine` and `trigger-aggregator`) runs on push. Revert before merge.
Write a summary table with branch, event, deploy, and allowlist status. Include a callout explaining why artifacts will or will not be published.
Intentionally not dropping original commits for transparency.
For `workflow_dispatch`, `github.sha` points at the launch ref (typically the default branch), not at the branch that `html-docs`/`spec-docs` actually check out via the `ref:` override. Published `metadata.json` and the `steel-website` dispatch payload therefore could reference a commit on a different branch than the one that was built. Resolve the tip SHA of the target branch in `check-should-publish` via `git ls-remote` and expose it as a job output. Consume that output in the `combine` metadata step and the `trigger-aggregator` dispatch.
The `edit_uri` value has no effect: mkdocs-material only renders the "edit this page" pencil when the theme feature `content.action.edit` is enabled, which this config does not enable. Drop the unused setting to avoid a stale reference to `forks/amsterdam` once docs are published for additional branches.
Previously the empty-allowlist branch wrote `should_publish=false` and exited 0, silently no-op'ing all downstream publishing. `exit 1` surfaces a misconfigured or missing `.github/configs/docs-branches.yaml` as a failed check instead.
Previously staged at `/docs/<branch>/spec/` as a top-level sibling of the mkdocs site. Nest under `/docs/<branch>/specs/reference/` so the URL mirrors the nav hierarchy (reference now lives inside the Specifications section). Add a "Reference" entry to the literate nav and update the home page card to link at the new path. Both use a ↗ glyph to signal the reference is a separate artifact. The home page link opens in a new tab via `attr_list`; the nav link does not, since `literate-nav` does not honor `attr_list`. Also add a placeholder at `docs/specs/reference/index.md` so mkdocs's strict-mode nav validation succeeds. At deploy time, the `docc` output overwrites the placeholder via `rsync`; locally it points users at `just docs-spec`.
Mkdocs treats bare directory URLs (`tests/`, `specs/reference/`) as potential external references and emits INFO-level "unrecognized relative link" messages, leaving the link text as-is. Suffixing with `.md` or `index.md` lets mkdocs resolve the link cleanly and emit the correct URL; the sibling cards in the home page already use this form. Fixes: - `tests/` -> `tests/index.md` in the home page card. - `specs/reference/` -> `specs/reference/index.md` in the home page card and the literate nav entry (now resolves against the stub page added alongside the path rename). - `../tests/prague/.../test_invalid/` -> `.../test_invalid.md` in `writing_tests/post_mortems.md`.
091e548 to
d346c9c
Compare
Add `experiments/**` branch trigger and `experiments/publish-docs` to the allowlist so the full pipeline (including `combine` and `trigger-aggregator`) runs on push. Revert before merge.
Material's `navigation.instant` (`mkdocs.yml:51`) intercepts internal nav
clicks and XHR-swaps the target page into the current DOM. The
`docc`-rendered reference at `/specs/reference/` has no Material template
markers, so the swap silently fails: the URL and `<title>` update to the
reference page but the body stays on the original. Clicking "Reference"
from the sidebar or the footer next/prev buttons on `/specs/` appears to
do nothing.
The home page card already opens in a new tab via `attr_list`
(`{target=_blank rel=noopener}`), which makes instant-nav opt out and do
a full-page load. `attr_list` doesn't carry through `literate-nav` or the
footer template, so add an `on_post_page` hook that rewrites any rendered
anchor pointing at the reference (`reference/`, `../reference/`,
`specs/reference/`, `../../specs/reference/`, etc.) to include
`target="_blank" rel="noopener"`, skipping links that already have the
attribute set.
Revert the `experiments/**` branch trigger and the `experiments/publish-docs` allowlist entry added in `6208a3b775a` for E2E testing. Intentionally not dropping original commits for transparency.
Let maintainers publish or preview docs for any tagged/branched commit
without waiting for a push to an allowlisted branch. Dispatch now takes:
- `branch` (string, required, default `forks/amsterdam`): Branch to
publish under. Must be in the allowlist at
`.github/configs/docs-branches.yaml`; `check-should-publish`
validates and prints the allowlist into the job summary on mismatch.
Kept as `type: string` rather than `type: choice` so the allowlist
YAML stays the single source of truth; adding a branch is a
one-file edit.
- `ref` (string, optional): Branch, tag, or commit SHA to build. Empty
falls back to the branch tip. Resolved to a concrete SHA up front
via `gh api repos/{owner}/{repo}/commits/{ref}`, so `metadata.json`
and the aggregator payload always reference the commit that was
built.
- `publish` (boolean, default true): When false, `html-docs` and
`spec-docs` still run and upload their per-job artifacts, but
`combine` and `trigger-aggregator` are skipped. Use for dry-run /
artifact inspection without deploying.
`html-docs` and `spec-docs` now check out the resolved SHA rather than
the branch name, which lines up the build jobs with the SHA recorded
in metadata and dispatched to steel-website.
The job summary also echoes the expected deploy URL on a publishing
run, with a note that resolution is gated on steel-website's own
`BRANCH_CONFIG` (`deploy.yml`).
Adds `devnets/bal/4` (label `bal-devnet-4`) to the publish allowlist and the site's version switcher so pushes to that branch publish at `/docs/devnets/bal/4/`.
664445a to
3496b74
Compare
The `publish` boolean on `workflow_dispatch` was intended as a dry-run (build without deploying), but PRs already exercise the build path and local `just docs`/`just docs-spec` cover ref-specific builds. It was also silently broken: the upload steps gated on `is_deploy == 'true'`, so `publish=false` produced no artifacts despite the summary claiming otherwise. With `publish` gone, `is_deploy` becomes redundant with `should_publish` on every gate, so collapse both into a single `should_publish` check across the upload steps, `combine`, and `trigger-aggregator`. Drop the `Deploy build` row from the check-should-publish summary.
The spec-docs `rsync` targets `stage/${BRANCH}/specs/reference/`,
but the preceding `mkdir -p` still created `stage/${BRANCH}/spec`
(singular, leftover from the pre-`specs/reference/` layout). It
worked by accident because the prior `rsync -a html-docs/` copied
mkdocs's `specs/` subtree into the stage, so the intermediate
directory happened to exist by the time the spec-docs rsync ran.
Create the exact target path up front so the staging step no
longer depends on mkdocs output shape.
`steel-website` serves docs for multiple repos under `steel.ethereum.foundation/docs/<repo>/`. Prefix every URL we publish or reference with `execution-specs` so the built site, dispatched artifacts, and documented links resolve under the correct namespace. Updated: - `mkdocs.yml` `site_url` fallback. - `DocsConfig.DOCS_BASE_URL` (ruff-formatted onto its own line). - `README.md` docs link. - `docs/dev/docs.md` hosting URL. - `.github/workflows/docs-build.yaml` site-structure comment, `Will publish` job summary URL, and `SITE_URL` export. - `.github/configs/docs-branches.yaml` comment header.
The field was never read by the workflow (which only consumes `.branches[].path`) and doesn't mirror anything on `steel-website` either: its `docs-config.yml` uses a different key (`default_branch:`). Remove the dead entry and tighten the sync-comment so it names the actual source of truth (`docs-config.yml`) and the specific invariant (the `branches[]` list).
`check-should-publish` was always resolving the commit SHA via `gh api repos/.../commits/<branch>`, which on push returns the branch tip at API-call time. If another push lands in the gap between the event firing and the API call, `commit_sha` (written into `metadata.json` and the `steel-website` aggregator dispatch) drifts from the actual commit the build jobs check out: the build jobs fall back to `github.sha`, i.e. the pushed commit. Use `$GITHUB_SHA` directly on push so every artifact and the downstream dispatch references the same commit. Keep the `gh api` resolution for `workflow_dispatch`, where `github.sha` points at whichever ref the dispatch was launched from rather than the user-specified `branch`/`ref`. Concurrency cancellation already shrinks the race window, but this closes it cleanly.
- `docs/dev/docs.md`: strip the "(soon) hosted remotely on Github Pages" phrasing. The site is now served by `steel-website` under `steel.ethereum.foundation/docs/execution-specs/`. - `docs/specs/index.md`: point the "Rendered specification" entry at the new `specs/reference/` path instead of the `ethereum.github.io/execution-specs/` gh-pages URL. - `docs/specs/writing_specs.md`: retarget the fork "diff outputs" link to the new `specs/reference/diffs/` path.
Add `experiments/**` to the push trigger and `experiments/publish-docs` to the allowlist so a push to this branch exercises the full pipeline (`combine` and `trigger-aggregator` included) before the PR is marked ready. Revert before merge.
Revert the `experiments/**` branch trigger and the `experiments/publish-docs` allowlist entry added in `cc7c0654d86` for E2E testing. Intentionally not dropping the original commit for transparency.
kclowes
left a comment
There was a problem hiding this comment.
This looks awesome! I poked around the docs a bit, and they look great! I just left a comment that was more of a question for my own knowledge/understanding, but nothing blocking at all.
And just in case you haven't seen: https://www.reddit.com/r/Python/comments/1s0gfyb/the_slow_collapse_of_mkdocs/. I don't think there is anything to do right now, but worth keeping an eye on.
Restrict the `push:` trigger to `mainnet` and `forks/amsterdam`. Other allowlisted branches in `.github/configs/docs-branches.yaml` now publish only via `workflow_dispatch` (use the `ref:` input to publish a tag/SHA). The allowlist remains the source of truth for which branches are allowed to publish; this change separates "allowed to publish" from "auto-publishes on push".
Add a `mainnet` entry to `.github/configs/docs-branches.yaml` with label `Mainnet (BPO2)` so pushes to `mainnet` (already in the workflow's auto-publish trigger) pass the allowlist gate. Requires a matching entry in `steel-website`'s `BRANCH_CONFIG` for the URL to actually serve.
Thanks @kclowes, yes, I saw this (painful story!), it's one reason the steel-website uses zensical, which is a promising replacement for mkdocs. The blog comments that zensical is the most popular and seems the most likely successor atm. But we should form our own opinion. We should aim to migrate away from mkdocs-material before EOY as critical bug fixes and security updates will only be made until Nov 2026 (squidfunk/mkdocs-material#8523). |
Summary
Add a
docs-build.yamlworkflow that builds HTML docs (mkdocs) and spec docs (docc) in parallel, combines them into a single artifact, and dispatches thesteel-websiteaggregator for deployment tosteel.ethereum.foundation/docs/execution-specs/.docs-build.yaml:test-docs.yaml(removed in this PR).gh-pages.yaml(follow-up cleanup tracked in Add redirect from execution-specs gh pages to steel-website docs/spec and deletegh-pages.yaml#2684), we should add a one-time redirect from the old gh-pages site to the docc spec on steel.ethereum.foundation, once this lands.Changes
Workflows:
.github/workflows/docs-build.yamlwith five jobs:check-should-publish,html-docs,spec-docs,combine, andtrigger-aggregator. Also adds aworkflow_dispatchentry point withbranch(required, must be allowlisted) andref(optional: branch, tag, or SHA to build) inputs for manual runs.push:trigger only fires formainnetandforks/amsterdam(auto-publish set). Other allowlisted branches in.github/configs/docs-branches.yamlpublish only viaworkflow_dispatch(use theref:input to publish a tag/SHA). A follow-up will move this policy into the allowlist YAML (publish_on_push/publish_on_tagflags + default-branch sentinel)..github/workflows/test-docs.yaml; move itslint-mdandchangelogjobs intodocs-build.yaml.Config:
.github/configs/docs-branches.yaml: branch allowlist controlling which branches are allowed to publish (auto-publish on push is gated separately by the workflow'spush: branches:filter).mkdocs.ymlsite_urlto read from theSITE_URLenv var, with fallback tosteel.ethereum.foundation/docs/execution-specs/.DocsConfigbase URL tosteel.ethereum.foundation/docs/execution-specs.Docs:
docc-rendered reference under/docs/<branch>/specs/reference/to mirror the nav hierarchy.docs/hooks/reference_new_tab.py: force reference nav links to open in a new tab (works around thenavigation.instantXHR-swap replacing the whole site).docs/specs/reference/index.mdplaceholder so mkdocs strict-mode nav validation succeeds locally; at deploy time thedoccoutput overwrites it viarsync.Trigger matrix
checkhtml-docsspec-docscombinetrigger-aggregatorpull_requestpushtomainnetorforks/amsterdampushto any other branchworkflow_dispatch(allowlisted branch, optional ref)Validation
Successful publish here:
Related Issues or PRs
gh-pages.yaml: Add redirect from execution-specs gh pages to steel-website docs/spec and deletegh-pages.yaml#2684.Checklist
just statictype(scope):.mkdocs servelocally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.Cute Animal Picture